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CORRELATING CAPTURED IMAGES AND TIMED 3D EVENT DATA 



CROSS-REFERENCE TO RELATED APPLICATION^) 

Reference is made to copending application Serial No. 10/273,871, 
5 entitled "Correlating Asynchronously Captured Event Data and Images" and filed 
October 18, 2002 in the names of M. A. Valleriano, C. I. Marshall and M. A. 
Bobb, and copending application Serial No. 10/303,363, entitled "Correlating 
Captured Images and Timed Event Data" and field November 25, 2002 in the 
names of M. A. Valleriano, C. I. Marshall, and M. A. Bobb, which are assigned to 
10 the assignee of this application. 

FIELD OF THE INVENTION 

The invention relates generally to the field of digital image 
processing, and in particular to the automatic correlation of images with objects in 
15 the images or events associated with the images. 

BACKGROUND OF THE INVENTION 

There are a number of ways to identify a particular person within 
an image, picture or photo. One typical method provides the person with an 
2 0 identification number, and that identification number is then associated with an 

image. A few examples of such methods include magnetic stripe cards, bar codes, 
and radio frequency identification tags that are encoded with the person's 
identification number. The person's identification number is read before, during 
or after the image capture and the identification number is associated with the 

2 5 specific image by known methods (e.g., encoding the identification number in the 

image's metadata or recording the identification information in a database). 

Eastman Kodak Co. has a number of products that associate a 
particular person with an image. For example, Kodak EPX Thrill Shots and 
Roving Photos, Kodak Image Magic Fantasy Theater and other Kodak products 

3 0 provide the subject with an identification (ID) tag that is associated with an image 

and used to find the image in an image database and produce a photographic 
product. 
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Just as important, these systems are difficult to apply in a situation where the 
identifier codes are not specifically tied to a particular image, that is, the identifier 
codes and the images are obtained by systems that are not necessarily 
synchronized (i.e., asynchronous). 
5 The aforementioned related Patent Application Serial Nos. 

10/273,871 and 10/303,363 describe methods for identifying people in an image 
using a simple location detection system based on a single data reading device per 
image location, and especially where the images are asynchronously captured 
relative to event data associated with a timed activity. For example, a race may 
10 have a timing mat at a specific location on the race course. As a contestant 

crosses the timing mat, data from an RFID tag is read by the mat and a data record 
is created. The resulting information is basically two dimensional (2D) in nature 
(i.e., the mat location on the course), and the location of a contestant when data is 
read by the timing mat is essentially fixed throughout the race since the mat is not 
15 relocated during the race. Meanwhile, a camera is capturing images of the event 
(e.g., contestants crossing the finish line), but the images are not directly 
associated with the timed activity (i.e., the images are captured asynchronously in 
relation to the data record from the timed activity). A correlation algorithm then 
relates the images to the data record, and thereby to the RFID tag information 
2 0 identifying the people in the images. 

However, there are certain situations where the location of the 
person being photographed can vary across a large volume of space. For 
example, in a sporting event such as a hockey, football or soccer game, players 
may be anywhere on the field or on the periphery of the field. Images of these 
25 players may be taken while they are anywhere within the sports field or arena. 

This situation requires a three dimensional (3D) tracking system to locate players 

during a sporting event. 

One three dimensional tracking system for a sports event is 
described in WO 00/2873 1 (Orad Hi-Tec Systems Limited). This document 
30 describes an interactive television system using video and/or television cameras to 
image a sporting event, such as a game. RF or microwave sensing equipment is 
used to identify players and objects, and triangulation is used to determine the 
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inc.ud.ng the identities of the players, is asynchronously captured in relation to 
fire high quali.y images, such ma, there is seidom a dear one-to-one relanonshtp 
between the event data and the images. 

SUMMARY OF THE INVENTION 

The present invention is directed to overcoming one or more of the 
problems se, forth above. Briefly summarized, accordmg to one aspect of the 
present invention, a method for correlating tracking data associated w„h an 
activity occurring in a rhree-dimensional space with images captured wtthm 
space, wherein remotely-accessible identification tags are attached to people 
and/or objects associated with the acfivity, comprises the steps of: (a, 
camera with respect to the three-dimension* space, wherein the camera a, a gtven 
iocation has a definable orientation and field of view that encompasses at least 
a portton of the space; (b) capturing a p.urahty of images with the camera and 
srormg data corresponding to the Images, including a capture time for each , mage, 
(e) capturing tracking data from the identification tags attached to the people 
and/or objects within the space and storing .he tracking data, includmg a fag 
capmre time for each rime fha, a «, is remotely accessed; (d) correlating each 
image and the tracking data by interrelating hacking data having a tag capture 
fime in substantial correspondence with the capture time of each image thereby 
generating .rack data corresponding ,o each .mage; (e) utilizing the hack data ,o 
determine positions of the people and/or objects wtthm the three d.mensrona, 
space a. the caprine time of each image; and (f) utilizing the location and 
orientation of the camera to determine the portion of ,he space captured m each 
5 image and thereby reduce me hack dafa to a hack dafa subset correspond™, ; ,o 
people and/or objects positioned within the portion of space captured ,n each 

taaSe ' The advantage of the invention lies m its ability to easily correlate 

asynchronous* captwed even, data and images in a three dimensmna, volume.™ 
,0 space on a playfng field. As a result, in a sporting even, occurring on a playmg 
field it is possible ,o easily idenfify and associate all the players wtthm a 
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particular image of the playing field, and also to easily locate all images of the 
playing field that include a particular player. 

These and other aspects, objects, features and advantages of the 
present invention will be more clearly understood and appreciated from a review 
5 of the following detailed description of the preferred embodiments and appended 
claims, and by reference to the accompanying drawings. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a pictorial illustration of a playing field, wherein a 
1 0 photographic and tracking system is used according to the invention to capture 
and identify images of players that are within a defined volume or space. 

FIG. 2 is a pictorial illustration of a tracking system that is useful 
in the playing field embodiment shown in Figure 1 . 

FIG. 3 is a pictorial illustration of a plurality of playing fields 
15 covered by one or more tracking systems. 

FIGS .4 A and 4B show two configurations of a digital camera used 
in the playing field embodiment shown in Figure 1, one (4A) in which tracking 
data is stored with the image and one (4B) in which tracking data is wirelessly 
transmitted to a remote computer. 
2 0 FIGS . 5 A, 5B and 5C shown three embodiments for tag placement 

with a digital camera, including a first (5 A) with RFID tags on the camera body, a 
second (5B) with RFID tags on a single axis bar supporting the camera, and a 
third (5C) with RFID tags on a two-axis bar supporting the camera. 

FIG. 6 shows a camera mounted on a mechanized gimbal for 

2 5 determining line of sight data. 

FIGS. 7 A and 7B show two approaches to implementation of the 

invention. 

FIG. 8 is a diagram of the various databases used in the playing 
field embodiment and their interrelationships. 

3 o FIG. 9 is a diagram showing various locations for storing the 

camera tracking information. 
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FIG. 10 is a pictorial illustration of two playing fields showing the 
use of depth of field to reduce the amount of tracking data. 

FIG. 1 1 is a block diagram of the basic functions performed 
according to the present invention. 
5 FIG. 12 is a block diagram showing further details of the 

preparation function shown in Figure 1 1 . 

FIG. 13 is a block diagram showing further details of the capture 
function shown in Figure 1 1 . 

FIG. 14 is a block diagram showing further details of the tracking 
10 function shown in Figure 1 1 . 

FIG. 15 is a block diagram showing further details of the 
correlation and data reduction function shown in Figure 1 1 . 

FIG. 16 shows an illustration of how camera time is used to relate 
the track time for given tag ID to images. 
15 FIG. 17 is an illustration of an image file including a header data 

portion and an image data portion. 

FIG. 18 is an example of a listing of data for selected tag IDs 
obtained from data stored in the databases. 

FIG. 19 is a block diagram showing further details of the preview 
2 0 and print function shown in Figure 1 1 . 

FIG. 20 shows the interrelationship of the data in several of the 

databases. 

FIG. 21 is a pictorial diagram of a computer system for 
implementing the present invention. 

25 

DETAILED DESCRIPTION OF THE INVENTION 

Because image processing systems employing correlation and 
retrieval techniques are well known, the present description will be directed in 
particular to attributes forming part of, or cooperating more directly with, a 
30 method and system in accordance with the present invention. Method and system 
attributes not specifically shown or described herein may be selected from those 
known in the art. In the following description, a preferred embodiment of the 
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present invention would ordinarily be implemented as a software program, 
although those skilled in the art will readily recognize that the equivalent of such 
software may also be constructed in hardware. Given the method and system as 
described according to the invention in the following materials, software not 
5 specifically shown, suggested or described herein that is useful for 

implementation of the invention is conventional and within the ordinary skill in 
such arts. If the invention is implemented as a computer program, the program 
may be stored in a conventional computer readable storage medium, which may 
comprise, for example; magnetic storage media such as a magnetic disk (such as a 
10 floppy disk or a hard drive) or magnetic tape; optical storage media such as an 
optical disc, optical tape, or machine readable bar code; solid state electronic 
storage devices such as random access memory (RAM), or read only memory 
(ROM); or any other physical device or medium employed to store a computer 
program. 

15 The advantage of correlation, as described in the cross-referenced 

Serial No. 10/303,363, is understood with the example of a foot race. Races such 
as a marathon or ironman race typically have many contestants. Photographs of 
the contestants may be taken at the beginning of the race, at the finish line, or at 
any point along the race course. These photos may show a few people, but 

2 0 typically contain many people. Also, many photos may be taken throughout the 
race, and a given person could appear in a number of these photos. Similar 
situations occur at other events such as graduations, walks for charity, and so on. 

At a race, the contestant initially registers by providing personal 
information that is stored in a registration database. The contestant is issued a 

25 Contestant ID number (CID) that is recorded in the registration database. The 

CID is also provided on a bib, badge, pass or other article that the contestant can 
carry or wear. In the prior art, for example, the article could also contain an RFID 
tag. Consequently, these articles include the CID and a unique tag article ID 
(TID) number. Information relating the CID and the TID is recorded in an article 

30 database (which could be different from the registration database). 

For instance, the contestant is provided with a bib having a CID 
and an RFID tag with the TID; optionally, the RFID tag may also have the CID. 
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It is possible for the RFID tag to be incorporated into the bib, or it might be a 
separate item such as a wrist or ankle band for wearing. 

Data gathering stations are located at one or more points around 
the race course. Each data gathering station includes a means (e.g., timing mats 
5 laid across the course) to read information from the RFID tag (or other article) as 
the contestant passes the station, and a way to associate that information with 
other data such as time, location, lap, etc. that is obtained from separate race 
timing apparatus. This information is stored in a race time database. The data 
gathering station may also include at least one camera that captures one or more 

10 images as the contestant races past the station. Ordinarily, the camera associates 
data such as time of capture, image number, camera number, etc., with the image 
in a camera image database. The challenge is to correlate the information from 
the various databases using the CID, TID, time, location and other data. 

In cross-referenced Serial No. 10/303,363, a method is described to 

15 associate all the contestants within a given photo with that particular image, and 
also to locate all images that include a particular contestant. Such a method is 
particularly useful in a system where the data about the event, including the 
identities of the contestants, is asynchronously captured in relation to the images 
such that there is seldom a clear one-to-one relationship. To provide correlation, 

20 a time-based window is generated that interrelates event data in the race time data 
base with images in the camera image database having a time of capture within 
the window. Once that relationship is established between the event data and the 
images, the race time data base can relate the images to the identification of the 
race contestants that would likely be in the images for that segment of the event. 

2 5 Referring first to Fig. 21 , there is illustrated a computer system for 

implementing the present invention. Although the computer system is shown for 
the purpose of illustrating a preferred embodiment, the present invention is not 
limited to the computer system shown, but may be used on any electronic 
processing system such as found in personal desktop or laptop computers or 

3 0 workstations, or any other system for the processing of digital images. The 

Kodak EPX system, a professional image operating system offered for the 
entertainment industry, serves as the basis for the photographic system used in 
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this embodiment. This system includes one or more cameras 310 (which are 
preferably either digital cameras capable of storing a considerable range of meta 
data (time, date, etc.) related to the captured images or film cameras capable of 
recording similar data on a magnetic region of the film (e.g., on APS film) or on 
5 an ancillary recording device). Images are captured by one or more of the 

cameras 310 and entered into the EPX system via removable storage media 312a 
(e.g., a Compact Flash card) or by a tethered (or wireless) link 3 12b between the 
camera 310 and a download computer 314. A point of consumer (POC) computer 
3 1 6 is used to find, preview, edit and select images for output. Images selected 

10 for output are processed and print queues are managed by a print server 3 1 8 for 
output to a printer 320 or to a poster printer 322. Note that all of these computer 
functions are coordinated by a network switch 324, but could be implemented on 
a single computer, or any other number, or combination, of computers as required. 

Additional computers (not shown) could be added to the system for 

15 additional functions. For example, a preview computer could be used to display 
images on preview monitors to entice players and other customers to the sales 
location. A dedicated operator computer could be provided for backroom 
operations. Other computer system architectures are possible, and many have 
been implemented in Kodak EPX installations. Relative to a subsequent 

2 0 description of the present invention, the computer system shown in Figure 2 1 is 

referred to as the imaging system 82, which interrelates via an arrow 84 (which 
may comprise software) with a tracking system 80. 

Unlike a race, where contestants follow a defined path that 
intersects timing mats in defined locations, images may be captured at an event 
25 where people and objects travel throughout a defined volume of space. Such an 
event may be a sports event such as a soccer, hockey or football game, or it may 
be any gathering of people within a defined space, such as an assemblage of 
people for a meeting (e.g., a political convention). 

Figure 1 shows a playing field 8 within which players within a 

3 0 defined space are photographed according to one embodiment of the instant 

invention. A sports event such as a soccer game is typically played on such a 
defined field. The field typically includes a boundary 10 within which the game 
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is played by a group of active players 12, and outside of which non- active players, 
coaches and others on-lookers 14 may congregate. The field often includes 
locations of special interest, such as a goal post 16, and team bench or penalty box 
18. 

5 An objective of the present invention is to provide identification of 

all players, e.g, players C, D, E, F and G, that are within an image captured by a 
camera 20 (in one embodiment, the camera 20 is one of the digital cameras 310 
shown in Figure 21). Such identification information is stored in a database 
environment. In the following description, and as shown in Figures 8 and 20, 

10 database information will be referred to as existing in separate databases such as a 
tag database 100 or a participant database 102. It is well known to those in the 
database arts that such independent databases could be variously located in either 
the imaging system 82 or the tracking system 80, or can be combined or 
distributed in many alternative combinations between or apart from these systems, 

15 including one large database in one venue. All of these alternative combinations 
are included within the scope of this invention. 

Within an existing and conventional tracking system, information 
about some or all of the players, coaches, referees, on-lookers, photographers, and 
the like is entered into a participant database 102 (see Figure 8). For players in 

2 0 particular, this information may include biographical data, player performance 

statistics, participant ID, and any other data of interest. Information about objects 
of interest (e.g., a goal, a penalty box or a camera) may be entered in the 
participant database 102 or in an optional other features database 104 (see Figure 
8). Object information may include a description, "participant" or camera or 

25 feature ID, etc. If the object has a fixed location (such as along the boundaries of 
the field), this database may include information on the object's location (based 
on XYZ coordinates), perimeter location, etc. 

As explained heretofore, participants (and objects of interest) have 
an article that identifies them uniquely through a non-contact communication 
30 method (e.g., an RFID tag, etc.). For instance, a tag 22 (see Figures 1 and 2) is 
provided to players and objects on a bib, badge, pass or other article that a player 
or other object may carry or wear so that their locations during the event may be 



tracked, and the appropriate tag identification information is entered into the tag 
database 100. Each tag 22 contains a tag identification (tag ID) that is unique for 
each player and each object. Information entered into the tag database 100 (see 
Figure 8) thus relates the tag ID to a participant ID, a camera ID, a feature ID, etc. 
5 These tags may utilize any tracking technology known to those skilled in the art, 
such as RF, microwave, optical, magnetic, etc. RF tags may be active or passive 
in nature. For the purposes of this description, we will assume active RF tags are 
used. A tracking system 21 employing an active RF tag 22 is shown in Figure 2; 
wearable tags of this type are well known; for example, see the tag described in 
10 International Publication Number WO 00/16493, which was published 23 March 
2000. 

One or more tracking stations 24 are located around the periphery 
of the field 8 shown in Figure 1 . As shown in more detail in Figure 2, these 
tracking stations 24 may include one or more antennae, e.g. a receiving antenna 

15 26 and a transmitting antenna 28, an electronics portion 30 that may include a 
transmitter and/or a receiver (sometimes referred to as a transceiver 30 or a 
transponder), and a means to communicate to a remote computer 32 including an 
external database 34 networked through a gateway 36 to a server 38. The 
communication capability may comprise a wireless transmission 40 between the 

2 0 transceiver 30 and the server 38. Other communication capabilities may be 

employed, such as a tethered connection between the electronics portion 30 and 
the server 38. Active tags 22 transmit data periodically, or on command. Tag 
data may include a unique identifier (tag ID) as well as other information, such as 
battery power level. Data from the tags is received by the tracking stations 24, 

2 5 where it is recorded with information about the time the data was received. The 

timing data may be provided by the electronics portion 30, which for this purpose 
includes a clock, or access to a clock, or the timing data may be provided by the 
remote computer 32. 

Referring to Figure 8, received tag data including the tag capture 

3 0 time is typically stored in a tracking database 106. The tracking database may be 

distributed among the tracking stations 24 or it may be centralized in the remote 
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computer 32. Data from the one or more tracking stations 24 is typically sent to a 
remote computer via a network (e.g., the wireless network 40). This embodiment 
will describe a wireless network, although other methods known to the art may be 
used, such as a wire cable, an optical cable, etc. 
5 The remote computer 32 then processes the tracking data to 

determine the location of each tag 22, and the player or object that the tag is 
attached to, throughout the sports event. In conventional tracking system 
applications, the player or object information obtained from processing in the 
remote computer 32 is combined with the video feed from conventional TV 

10 cameras (which are not shown in Figure 1) which are televising the game. For 
example, information from the tag database 100 and the tracking database 106 
may be used to locate biographical information or individual statistics from the 
participant database 102 for display on a low image quality display (e.g., 
broadcast video quality display) such as a TV, video or Internet output device. 

15 Examples of such low quality (or motion) output are exemplified by video 
applications involving tracking systems provided by Trakus, Inc., Everett, 
Massachusetts. 

According to one embodiment of the present invention, one or 
more photographers 42 may be located anywhere within the defined volume of 

20 space comprising the playing field shown in Figure 1, or outside of but with 
optical access to the defined volume of space of the playing field 8 (e.g., a 
photographer 42 with a telephoto lens, as shown in Figure 1, located outside the 
defined volume of space of the playing field 8). These photographers are 
typically equipped with high quality camera equipment 20 (e.g., film cameras or 

25 high resolution digital still cameras) that is independent of the existing tracking 
system 24. Typically, the photographic output of the camera equipment 20 is 
processed and utilized independently of the contemporaneous television 
production, e.g., for newspaper coverage of the game. However, these 
photographers may have tags 22 and information entered into the participant 

30 database 102. 

Each camera 20 has a particular orientation determined by a line of 
sight 44 and a field of view 46, as shown in Figure 1 . The camera records the 
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images of all objects and players within the camera's field of view 46, as 
determined by the position of the line of sight 44. Typically, the camera 20 is a 
digital camera and ancillary image data may be captured and recorded with the 
digital image (although with certain film cameras, e.g., APS cameras, ancillary 
5 image data may be recorded on the film together with the image). Image data 
recorded with a captured image includes such information as camera ID, 
photographer ID, camera field of view, camera line of sight, lens focal length, 
exposure data, etc. The camera equipment also includes a clock for recording the 
camera time at the moment of image capture. Referring to Figure 8, this data is 

10 stored in the image database 108, which may exist as part of the EPX system 
shown in Figure 21, e.g., as part of the download computer 314. 

Camera line of sight data may be determined using any technology 
known to those skilled in the art. As shown in Figure 6, according to one such 
technology, the camera 20 is mounted to a tripod 72 and attached to a mechanized 

15 gimbal 70 that measures angle and azimuth readings, which is communicated to 
the camera for recording with the image data. According to another technology, 
as shown in Figures 4A and 4B, a tracking data acquisition unit 50 is attached to 
the camera 20, where the unit 50 can either send data to the camera for storage 
with the image via a data link 52 (Figure 4A) or transmit tracking data to a remote 

20 location via a wireless transmitter 54 (Figure 4B). In such a system, as shown in 
greater detail in U.S. Patent No. 6,292,215, the data acquisition unit 50 includes a 
vertical gyroscope (not shown here) and a directional gyroscope (not shown here) 
that are orthogonally mounted on a stable inertial platform relative to each other. 
The vertical gyroscope, aligned along an X or Y axis, measures yaw (rotation 

25 about the Z axis), while the directional gyroscope, aligned along the Z axis, 

measure both roll (rotation about the Y axis) and pitch (rotation about the X axis). 
It may also be possible to combine multiple tracking technologies, such as GPS or 
GIS technologies or position sensors on the camera, to determine where a camera 
is located and the direction of its line of sight. In each case, the positional data 

30 that is obtained from the various methods as recorded in an appropriate database, 
such as a separate camera position database 110 (Figure 8), which may exist as 
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part of the EPX system shown in Figure 21, e.g., as part of the download 
computer 314. 

According to another approach for determining the camera line of 
sight, as shown in Figures 5A, 5B and 5C, multiple RFID tags 22a and 22b may 
5 be attached to the camera 20 (Figure 5A) and used by the tracking system 80 to 
convert the three-dimensional location data obtained from the multiple camera 
tags to camera line-of-sight data. Another implementation of RFID tags is to 
attach multiple tags 22a, 22b and 22c to a fixture 56 which is in turn attached to 
the camera 20 as shown in Figure 5B. In this example, the angle 23 between the 

10 line connecting the tags and the camera's line of sight is recorded in one of the 
databases (i.e., either in the participant database 102, the other features database 
104 or the camera position 110 database). If the location resolution of the 
tracking system is sufficient, it may also be desirable to record the separation 
distance of the camera line of sight from one of the tags on the fixture, or it may 

15 be assumed that the camera line of sight is located equidistant between the tags on 
the fixture. The distance between the tags on the fixture is determined by the 
resolution of the tracking system. It may sometimes be desirable to provide at 
least three tags 22a, 22b, 22c on a 2-axis fixture 58 to obtain line of sight 
information in two planes of orientation (Figure 5C). 

20 As shown in Figure 9, camera tracking information 60 may be 

stored in a variety of locations. In some embodiments, this information 60 can be 
stored (storage step 62) with the image information in the image database 108. In 
another embodiment, camera tracking information 60 can be stored (storage step 
64) in the camera position database 110. In yet another embodiment, the camera 

2 5 tracking information 60 can be stored (storage step 66) in the tracking database 
106, or in other locations as desired. 

Examples of how the present invention can be implemented will 
now be discussed. There are two approaches to implementation of the benefits of 
the invention. As shown in Figure 7A, the tracking system 80 can be used to 

30 identify people and objects captured by the high quality imaging system 82, often 
as a precursor to providing high quality reproductions of selected images to 
parties interested in seeing certain people or objects. On the other hand, as shown 



-16- 

in Figure 7B, the high quality imaging system 82 may provide input images to the 
low quality viewing system, e.g., TV or the Internet, that is driven by the tracking 
system 80. The invention involves the capability to correlate the output of an 
existing tracking system 80 as represented in Fig. 2 with the output of an imaging 
5 system 82 represented by the camera 20 in Figure 1 and the system shown in 

Figure 21. As shown in Figures 7 A and 7B, this correlation capability is included 
either in the high quality image system 82 or as an upgrade to the tracking system, 
as represented by the tracking system 80, or as a separate intermediary process 
represented by the arrows 84. Any combinations or variations on these 

10 implementations are also part of this invention. For either of these 

implementations, Figure 1 1 shows the basic functions performed according to the 
invention, including a preparation function 120, a capture function 122, a tracking 
function 124, a correlation and data reduction function 126 and a preview and 
print function 128. Each of these functions will now be discussed in greater detail 

15 in regard to the respective implementations. 

First Implementation: Using tracking data to identify people and objects 
within an image 

In this implementation, and as shown in Figure 7A, information 
from the tracking system 80 is used to identify people or objects of interest within 
20 a high quality image provided by the high quality image system 82. This example 
involves a soccer game. 

Preparation function 

The preparation function 120 can occur anytime prior to the start 

of the game. As shown in Figure 12, the game preparation starts with the 
25 installation and calibration of tracking equipment in a tracking preparation step 
130 and the initialization of the tracking equipment in an initialization step 132. 
At the same or similar time, the high quality imaging system 82 is set up. First, a 
region(s) of interest is selected in a region selection step 136 and the high quality 
photo equipment is synchronized in a synchronization step 140. In a positioning 
30 step 142, the photo equipment is positioned so that the line of sight 44 generally 
aligns with the selected region of interest. Once the basic positioning and 
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initialization of the equipment is completed, the preparation activity turns to the 
population of the various databases in the population step 148. 

Prior to starting the game, information is entered in a participant 
data step 144 into the participant database 102, including such data as name, team, 
5 player number, player statistics, etc. Some information such as player statistics 
may reside in another database (such as a league statistics database) and could be 
linked to the participant database 102. A player ID may be assigned at this time 
or it may be entered when a tag 22 is assigned to a player. The participant 
database 1 02 may also include information on coaches, referees, photographers 

10 and other people for whom identification or location tracking is desired. 

The image database 108 contains camera ID and other information 
pertaining to image capture. The constant camera data, such as camera ID, may 
be entered in a camera data step 138. Then, as each image is captured, 
information is stored in the image database 108 such as camera ID, frame or 

15 image ID, camera capture time, exposure data such as shutter and iris settings, 

other camera settings such as flash activation and lens focal length, field of view, 
text description, etc. The digital camera 20 typically has a clock, and the camera 
clocks may optionally be synchronized in the synchronization step 140 to a 
reference clock prior to the game. There may be a separate image database 1 08 

2 0 for each game, or images from more than one game may be stored in a single 
database. In this later case, game or date information may also be stored. 

The other features database 104 with other features important to 
the game may be created as a separate database with data obtained from an other 
features data step 134, or may be combined with other databases. This database 

2 5 may include information useful for identification and location of important items 

such as the goal posts, field perimeter location, team bench location, penalty area, 
etc. This database may include such information as a feature ID, description, 
position and orientation data, perimeter location, etc. 

RFID tags 22 are assigned to players, coaches, referees, 

3 0 photographers and other people for whom identification or location tracking is 

desired. People with tags keep them on their person before, during and after the 
game to facilitate tracking and identification. Tags may also be assigned 
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(attached) to objects such as a goal post, ball, puck, or a camera. Accordingly, 
appropriate tag information obtained from a TAG entry step 146 is entered into 
the tag database 100 to relate the tags to the people or objects they are attached to, 
including such information as tag ID, participant ID, camera ID, feature ID, etc. 
5 While line of sight information may be obtained in a variety of 

ways, as already mentioned, in the instant example information about the line of 
sight 44 of a camera 20 is obtained using RFID tags on a 2-axis bar 58 attached to 
the camera (see Figure 5C). The tracking system obtains location information 
from the tags and computes the camera orientation and line of sight from this 
10 information, which is stored in the tracking database 106 (or the camera position 
database 1 10). The camera field of view 46 is contained in the camera settings 
information stored with each image in the image database 108. 

Image Capture Function 

Referring to Figure 13, the capture function 122 begins prior to the 

15 start of the game (starting step 150), and continues until all images have been 
captured. Prior to the start of the game, participants obtain the tag 22 and any 
additional information about the participants may be entered into the appropriate 
databases. The tags 22 are attached to clothing, worn or otherwise retained during 
the sports event. Tags are also attached to objects of interest and corresponding 

20 information is entered into the appropriate databases. 

Photographers may capture images (capture step 1 52) at any time 
after tags are attached to the camera 20 and the tracking system 80 is functional. 
Captured digital images may be stored in a temporary image database in the 
camera (e.g. removable memory), together with certain camera data — in particular 

2 5 including the capture time. Captured film images are stored on the film, which is 

subsequently scanned to provide digital images, and the camera data is stored on 
the film or on an ancillary storage device. The images and the camera data are 
transferred from the camera (or scanner) in a transfer step 154 and stored in a 
storage step 1 56. In one embodiment, the removable memory is periodically 

3 0 removed from the digital camera and placed in a remote computer where the 

images are transferred to a primary image database. Alternatively, in another 
embodiment, the digital camera may transmit the images by a cable connection to 
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a remote computer for storage in the primary image database, or in yet another 
embodiment they may be transmitted by a wireless communication system to a 
remote computer for storage in the primary image database. The instant example 
will describe cameras with wireless connections to a remote computer. 

5 Tracking Function 

Referring to Figure 14, after the sporting event starts (starting step 

160), the tracking system shown in Figure 2 detects TAG data on the playing field 
in a TAG data detection step 1 62 and processes the TAG data relative to its time 
of arrival in a processing step 164 to extract the time of arrival for the tagged 

10 event. The now-timed TAG data is next processed in a signal processing step 166 
to obtain positional data of the tagged persons or objects for the given time. 
Finally, the processed data is stored with time and positional coordinates for later 
usage in a TAG data storage step 168. 

The operation of a tracking system such as shown in Figure 2 is 

15 well known to those skilled in the art. Generally, the tags 22 transmit unique 

identifying data (periodically or on command). The antenna 26 receives the tag 
data, which is processed and stored in the track database 106. The processed tag 
data may include track time, position and orientation data in any format such as 
XY, XYZ, XYZ-Roll-Pitch-Yaw, vector representation, etc. Additional 

2 0 information may also be stored, such as accelerometer data, etc. The track 

database 106 may also include one or more relative time values in addition to the 
track time. For instance the game time, the actual world time (e.g. Greenwich 
time), penalty time, or the time for each television camera used may also be stored 
in the track database 106. 

25 A conventional tracking system, such as the one described in U.S. 

Patent No. 6,204,813, features a radio frequency (RF) positioning system that 
determines the identity and positional data such as location, velocity, and 
acceleration of numerous objects. The system includes a plurality of spread 
spectrum radio transceivers where at least one transceiver is positioned on each 

30 object. Using spread spectrum radio transceivers is advantageous because it 
allows unlicensed operation. 



-20- 

At least three spread spectrum radio tower transceivers transmit to 
and receive signals from the plurality of radio transceivers on the objects. The 
three spread spectrum radio tower transceivers may employ directional antennas. 
Also, a processor is electrically coupled to each spread spectrum radio tower 
5 transceiver. The processor determines the time of arrival of signals received by 
each spread spectrum radio tower transceiver. 

Each of the tower transceivers are coupled to a central signal 
processor by a network. The central signal processor determines the identity and 
positional data of the objects. The signal processor determines at least one of: 
10 position; time derivatives of position; orientation; and time derivatives of 

orientation. The signal processor may be connected to the spread spectrum radio 
transceivers by any network, such as an Ethernet, fiber optic or wireless network. 

The central signal processor includes a memory to store the 
identity and the positional data of the objects. A video processor may be used to 
15 display the identity and the positional data of the objects on a video display 

terminal. In addition, the central signal processor may include a database engine 
for storing and retrieving data relating to the objects. The data may include 
biographical data of players in a game such as physical characteristics (height, 
weight, and strength and speed metrics) and previous game statistics. The video 
2 0 processor can display the data relating to the objects separately or together with 
the identity and the positional data of the objects. 

Such conventional systems also feature a method of determining 
identity and positional data of numerous objects in a three-dimensional space. 
The method is based on transmitting a spread spectrum signal that instructs a 

2 5 particular one of the plurality of spread spectrum radio transceivers on the objects 

to transmit a signal that can be processed to determine identity and positional data 
of the transceivers. 

Signals are received from the spread spectrum radio transceivers 
on the objects with the spread spectrum radio tower transceivers. The signal 

3 0 processor coupled to the spread spectrum radio tower transceivers despreads the 

signals to determine the identity of the objects, and processes the signals to 
determine the positional data of the objects. The positional data may be at least 



one of: position; time derivatives of position; orientation; and time derivatives of 
orientation. The positional data of the objects may be determined from estimates 
of the times of arrival of the signals to each of the (at least) three tower antennas. 
The times of arrival may be measured relative to a synchronization clock. 
5 The method further includes storing the identity and the positional 

data of the objects. The method may also include displaying the identity and 
positional data relating to the objects on a video screen. Information specific to 
the objects may also be displayed on the video screen. Further information 
regarding such a tracking system may be found in the aforementiond U.S. Patent 
10 No. 6,204,8 13, which is incorporated herein by reference. 

Correlation and Data Reduction Function 

The next step is to determine who and what appears in each image. 

This is done through the correlation and data reduction function 126 using image 
data and track data. 

15 One implementation of the correlation and data reduction step is 

schematically shown in Figure 15. In the instant example, the camera location 
and orientation is determined by the tracking system 80 using tags 22 associated 
with the camera 20. In a data upload step 170, image data such as camera ID, 
image 10, camera time and other information such as field of view and depth of 

20 field are sent to the tracking system 80 for a selected image. Next, in a selection 
step 1 72, all tracking data is isolated and selected that corresponds to the uploaded 
camera time. Such tracking data includes tag ID, tag capture time and position of 
the tag at the tag capture time. The tracking system 80 determines the position of 
all tags in the tracking database at the camera time. If tracking data is not 

25 available at exactly the camera time, the tracking system uses the available 

tracking data that has a track time that is in substantial correspondence with the 
time of capture of the images to determine the position of the tags at the requested 
camera time. "Substantial correspondence" should be understood to include 
tracking data with track times that are within a variable range of the requested 

30 camera time. Such a range can be defined by a "window" and the tracking data 
can be examined with a window processor to identify track times that are in 
"substantial correspondence" with the time of capture of the images. 
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The tracking system 80 may next apply all, some or none of the 
following data reduction operations to the tracking data. 

In a physical limitations step 1 74, the tracking system 80 may 
apply a correlation and data reduction operation including known physical 
5 limitations to the tracking data. Physical limitations, typically defined as location 
data in the other features database 104, may include a region of interest (e.g., a 
high school sports facility), areas of interest (e.g., boundaries of soccer field #1, 
boundaries of soccer field #2, etc.), and features of interest (e.g., size and location 
of east goal, size and location of west goal, size and location of penalty box, etc.). 

10 A typical example of the two regions of interest (region of interest A and region 
of interest B) is shown in Figure 10, wherein each region is a separate playing 
field. Information relating to these physical limitations may be entered at any 
time prior to the game. The physical limitations may be applied to the tracking 
data prior to receipt of the image data, or at the time of a specific search or query 

15 as shown in Figure 15. Once the physical limitations are applied to the tracking 
data in the step 174, a physically limited subset of all track data is obtained. For 
example, a particular camera 20 might be limited to images taken at a specific 
soccer field, and the physically limited subset of track data for that camera would 
contain only track data for tags on the selected soccer field. 

2 0 The tracking system 80 may next apply a correlation and data 

reduction operation including optical limitations to the track data in an optical 
limitations step 176. This operation may be performed by the tracking system or 
by the imaging system 82. In the instant example, the tracking system performs 
this operation. Using the camera ID sent to the tracking system 80, the tracking 
25 system locates tag data associated with the camera ID and calculates the camera 
location, orientation and line of sight. The camera location, line of sight (LOS) 
44 and field of view (FOV) 46 information define the volume of space captured in 
the selected image (see Figure 1). The tracking system uses this camera line of 
sight (LOS) 44 and field of view (FOV) 46 information in the optical limitations 

3 0 step 1 76 to select only the data from the physically limited track data subset that 

fall within the camera's field of view, producing a FOV limited track data set. 
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An additional optical limitation operation may include the 
application of depth of field (DoF) information that may also be sent to the 
tracking system to further reduce the track data set. DoF information may be 
defined in several ways, such as a lookup table relating lens aperture and focal 
5 length. In the instant example as shown in Figure 10, the DoF is sent in the form 
of a maximum (114) and a minimum (116) tracking limit distance. The tracking 
system 80 applies this DoF information to the FOV limited track data subset to 
select only those tags 22 that are located within the FOV 46 between the 
minimum (116) and maximum (114) tracking limit distances from the camera 20. 

10 It is also possible to define these tracking limit distances from some other object 
than the camera, such as a goal or another tag (e.g., player). In yet another 
implementation, the focal distance of the camera for a given image may be used 
in conjunction with a proximity value, and the tracking system could compute all 
tags that fall within the FOV and are within the proximity value of the focal 

15 distance from the camera. Those skilled in the art can identify additional ways of 
defining an area within the field of view of the camera within which all tags are to 
be identified, and all such alternatives are included within the scope of this 
invention. The data at this point in the correlation and data reduction process is 
described as the DoF limited track data subset. 

20 In an inclusion and exclusion step 178, the correlation and data 

reduction process may also modify the selected track data by applying inclusions 
and exclusions. For instance, it may be desirable to always identify or include 
certain tags that otherwise would not be included in DoF track data subset. For 
example, a particular player may be of interest regardless of their location within 

25 the field of view of a given image, or an operator may be searching for all images 
with a particular player in the image regardless of where the player is located. 
Exclusions may also be applied to the DoF track data. For instance, it may be 
desirable to eliminate all game officials or photographers from the data subset. 

A selected track data subset is the result of performing all desired 

3 0 data reduction operations, where the selected track data subset includes at least 
the Tag ID, and optionally the position data, of all people and objects that are 
likely to be within the image after the data reduction operations. 



The next operation in the correlation and data reduction process is 
correlating the selected track data subset with biographic data (correlation step 
180) that is maintained in this example in the participant database 102. This 
operation can be performed by the tracking system 80 or by the imaging system 
5 82. The result is that the image system obtains, as shown in Figure 1 8, a listing of 
selected tags within a particular image, position and the biographic data 
associated with those selected tags, thereby providing identification of all selected 
people and objects within the image. 

The image system 80 may optionally store the tag and biographic 

10 information in the image database 108. This may be done using any database 
management method known in the art. In addition, or alternatively, the tag and 
biographic information may be stored with the image itself. As shown in Figure 
17, this is typically done in the image header portion 190 of the image file 188. 
The stored information may include all or a portion of the tag, biographic and 

15 position information available. The stored information may be optionally 

categorized (e.g. people within DoF; people within FOV (but not within the DoF); 
objects within FOV; etc.). One example of some of the stored information that 
might be included in the databases is shown in Figure 1 8 for a particular listing of 
Tag IDs. 

20 In the instant example, information such as the tag ID, position, 

camera ID and image or frame number are stored and categorized in the 
participant database for each identified person or object in the image. In addition, 
tag, position and biographic information of each identified person or object 
identified within an image may be stored and categorized in the image header 1 90 

25 of the image file 188. 

Note that while these correlation and data reduction operations 
have been described in a particular order, it is possible for the sequence to be 
organized differently and still provide the desired result. Alternatively, some of 
these operations may not be performed for a given image. Portions of the 

30 correlation and data reduction process may be performed at different times, and/or 
they may be performed on different computers, as desired. For example, the 
physical limitations portion 1 74 may be performed prior to the arrival of any 



-25- 

image data, or the physical limitations portion may be performed on the tracking 
system while the optical limitations 1 76 may be performed on the image system. 
(However, the optical limitation portion of the correlation process occurs at any 
time following image capture.) For example, referring to Figure 21, if the camera 
5 310 has removable memory 3 12a, the correlation process may occur in batches 
after the images are entered into the primary image database in the download 
computer 314. If images are transmitted from the camera 3 10 to the remote 
computer 314 via cable or wireless communication 312b, then the correlation 
process may occur immediately after the images are stored in the primary image 
10 database in the download computer 314. In the instant example, the correlation 

process occurs immediately after images are stored in the primary image database. 

This correlation and data reduction process may be repeated for 
each image of interest. In alternative implementations, a batch of images may be 
processed as a group. 

Preview and Print 

In a first preview and print scenario using the instant invention, the 

correlation and data reduction process 126 is performed when images are entered 
into the image database. Thus all images have had the identity of participants and 
objects determined and stored with the images in the image database 108. 

Referring to Figure 19, a customer (who may be a participant, 
spectator or other person) approaches an operator at a photo station, such as the 
image system 82 in Figure 21, and asks to see all images taken of a participant 
during the game (request step 200). An operator may identify the participant with 
information from the customer (e.g., name) or by reading the tag ID from the 
participant's tag, if available (information search step 202). The operator enters 
the information into the imaging system and performs a search for images of the 
participant. The imaging system determines if the correlation and data reduction 
process has been run on all images (query step 204), and if not, that process is 
performed (correlation and data reduction step 206) on any images without 
participant identification information. The imaging system uses the identifying 
information and other information to locate all images (search step 208) of the 
participant in the image database (see Figure 20). Alternatively, the participant 
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database could be searched. The located images are displayed for customer 
review (review step 210). The customer may edit images (edit step 212) and 
select images for output as prints, on a CD, or any other output method known to 
the art (output step 214). 
5 In a second preview and print scenario using the instant invention, 

the correlation and data reduction process 126 either has not yet been performed 
on all images, or the desired participant or object was not included in the final 
tracking data subset used to record identities in the image database. 

As before, a customer approaches a photo station to see images of 

10 a participant (request step 200). As shown in Figure 19, an operator uses 

information from the customer to perform a search for images of a particular 
participant (search step 202). First, the image system determines if the correlation 
and data reduction process has been performed on all images (query step 204). If 
true, the image system searches the image database for any images including the 

15 participant of interest (search step 208). Prior to display, the results of this search 
may be reduced according to appropriate criteria (e.g. participant distance from 
camera) and the results may be placed in a particular sequence for display based a 
preferred parameter (e.g., smallest distance from camera first). The customer or 
operator may choose to edit (edit step 212) any of the displayed images using 

20 techniques well known in the art. Images may then be selected for output (output 
step 214) in any desired method (e.g., prints, t-shirts, digital files on a CD, email, 
etc.). 

If the image system 82 determines that some of the images have 
not had the correlation and data reduction process 126 performed on them, then 

25 the image system sends appropriate information to the tracking system 80 (e.g., 
camera ID, camera time, etc.). The operator may also send additional limitations 
to the tracking system, including features of interest (e.g. sports field or game 
identifier). The tracking system then performs the correlation and data reduction 
process on those images. The tracking system returns the required data to the 

30 image system for those images, and the image system stores or updates the image 
database with the new data. The image system then proceeds to search the 
database for the particular participant requested by the customer, the selected 
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images are sorted and displayed, optionally edited and selected images ordered as 
before. 

Second Implementation : Locating high quality images for display on a low 
image quality tracking system 

In this implementation as shown in Figure 7B, information from a 
tracking system 80 is used to locate high quality images provided by a high 
quality image system 82 for display on the low quality video displays associated 
with the tracking system 80. 

Camera image data are stored in the image database 1 08 as 
described in the first implementation. Image data is correlated to data in the 
tracking database in a similar manner to that described in the first implementation. 
Images may be located and displayed or printed as described in the first 
implementation. 

The significant benefits of this implementation is that it enables 
independent photographers to identify players and objects within every image 
they capture, and the video, TV or Internet viewers can also access high quality 
(still) images and order products as desired. 

One implementation of the instant invention is for professional 
sports events such as a hockey game. Professional photographers can use the 
tracking information to identify players and puck locations in the photos they 
capture, and the sports news organizations can utilize the images in their 
broadcasts. Additionally, viewers can access and purchase images that interest 
them. 

Another implementation of the instant invention is for non- 
professional sports events such as a little league soccer game. A portable tracking 
system is set up around a sports field, and photographs are captured during the 
game. Players and audience members can purchase photographic products with 
the images, for either immediate or remote fulfillment. Other implementations 
could include less structured competitions, such as cheerleader competitions, or 
performances and performance competitions, such as band performances or band 
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festivals, where large numbers of individuals either perform in a defined three- 
dimensional space such as a stage or pass a defined staging location, such as a 
reviewing stand. Within an industrial or surveillance setting, the invention can be 
used to identify individuals within a three-dimensional controlled space, such as a 
5 factory floor or an entry foyer. 

Another implementation would provide a tracking system for more 
than one sports field (see Figure 3). In this embodiment, multiple sports fields 8a, 
8b, 8c, 8d may be covered by one or more tracking systems 21a,b. . . associated 
with the tracking stations 24. The tracking system can discriminate between 

10 separate fields or regions of coverage A, B, C and D by either having multiple 

tracking stations 24 for each field or by having a definition of regions of coverage 
within the tracking database 106. The tracking database 106 may also include 
region of coverage information within it. The correlation system would then 
eliminate players or objects within the field of view of a camera that was not also 

15 within the region of coverage of interest. 

In another implementation of the instant invention, attendees of a 
large convention (e.g. a political convention) or other gathering of people could 
all be provided with tags. A tracking system could be established around the 
convention site. For instance, assuming that the regions of interest A. . .D in 

20 Figure 3 represent different areas of a convention floor, the field of view of the 
camera 20 covering the region of interest D also extends over to the region of 
interest B. By using the aforementioned inclusions, exclusions and limitations, 
people or objects within the region of interest B may be eliminated from the field 
of view of the camera used for the region of interest D. Photographs captured 

25 during the convention would automatically be correlated to the tracking data to 
identify all those within a given image. 

The invention has been described with reference to a preferred 
embodiment. However, it will be appreciated that variations and modifications 
can be effected by a person of ordinary skill in the art without departing from the 

3 0 scope of the invention. 
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